A systematic literature review of methodology used to measure effectiveness in digital game-based learning

In recent years, a growing number of studies is being conducted into the effectiveness of digital game-based learning (DGBL). Despite this growing interest, however, it remains difficult to draw general conclusions due to the disparities in methods and reporting. Guidelines or a standardized procedure for conducting DGBL effectiveness research would allow to compare results across studies and provide well-founded and more generalizable evidence for the impact of DGBL. This study presents a first step in this process by mapping current practices through a systematic literature review. The review included peer-reviewed journal and conference publications between 2000 and 2012. Other inclusion criteria were that (1) the study’s primary aim was effectiveness measurement of cognitive learning outcomes, (2) the focus was on digital games and (3) a pre-post design with a control group was used. Twenty-five publications were found eligible for this study. Important differences were found in the number of control groups used and the type of intervention implemented in the control group (e.g. traditional classroom teaching, use of multimedia, computer-based learning, paper exercises, other games, or no intervention). Regarding the implementation method of the DGBL intervention in the experimental group, two approaches can be distinguished: stand-alone intervention or as part of a larger program. Moreover, a wide variety of effectiveness measures was used: measures for learning outcomes were complemented with time measurements and/or with self-reported measurements for self-efficacy and motivation. Learning effect calculation also varied, introducing pre-test scores in the analysis, conducting a separate analysis on pre- and post-test scores or conducting an analysis on difference scores. Our study thus indicates that a variety of methods is being used in DGBL effectiveness research opening a discussion regarding the potential and requirements for future procedural guidelines

Learning math as you play: comparing arithmetic performance enhancement induced by game play and paper exercises

One of the promises of video game training is that, compared to traditional training, it can be more engaging and entertaining (Boot et. al., 2008). However, besides entertainment, games have shown to have the potential to impact a larger variety of cognitive abilities. Previous research has consistently shown that several aspects in cognition such as visual short-memory, multitasking and spatial cognition can be enhanced by game play. In a previous study, we found that playing Monkey Tales, a commercial game aimed at training arithmetic skills in children, helped second grade pupils to increase their accuracy in mental calculation as compared to paper exercises or no exercises. The present study aimed to explore how arithmetic performance enhancement induced by game play and paper exercises differs. In order to do this, we compared the performance gains that second graders achieved in a computer test made for assessing their math skills. We performed a combined analysis of the changes in two behavioral measurements: accuracy and reaction times. Children were tested at two points in time: before and after the three week period. We compared the reaction times and the accuracy improvements between these two moments and compared different items types (e. g. understanding tenths, understanding hundreds, even or odd up to 100 among other types). We found indirect evidence suggesting that arithmetic performance enhancement induced by game play and paper exercises might rely on slightly different cognitive mechanisms

Pre-test session impact on the effectiveness assessment of a fire safety game

In recent years, critiques have been formulated regarding current evaluation methods of DGBL (digital game-based learning) effectiveness, putting the validity of certain results in doubt. An important point of discussion in DGBL effectiveness studies is whether or not a pre-test should be administered, as it can lead to practice effects and pre-test sensitization, threatening internal validity of the results. The present study aims at testing if the administration of a pre-test has a direct influence on post-test scores and/or makes participants more receptive to the intervention. For this purpose, an effectiveness study of a fire safety training in a hospital was conducted using a Solomon four-group design. The experimental groups received a game-based intervention (n= 65) of which 34 participants received a pre-test and 31 did not. The control groups received traditional classroom instruction (n=68), of which 39 participants received a pre-test and 29 did not. A 2x2 ANOVA was used to explore the practice effect and the interaction between the pre-test and the intervention. An interaction effect between pre-test and intervention is detected. More specifically, this interaction takes place in the traditional classroom group, indicating pre-test sensitization. In the traditional classroom context, the pre-test makes the participants more sensitive to the content treated in the intervention while administration of a pre-test does not influence outcomes of the DGBL treatment. When the administration of a pre-test influences the control group's receptivity to the treatment, but not the experimental group, results of an effectiveness study may be biased. This is especially relevant in the DGBL field as often, non-significant differences between DGBL and more traditional methods are reported. Therefore, further research should take this into account and look for possible solutions to solve this discrepancy

Changing your mind before it is too late: the electrophysiological correlates of online error correction during response selection

Inhibiting actions when they are no longer appropriate is essential for adaptive goal-directed behavior. In this study, we used high-density EEG and a standard flanker task to explore the spatiotemporal dynamics of cognitive control and inhibitory mechanisms aimed to prevent the commission of errors. By recording hand-related electromyographic activity, we could disentangle successful from unsuccessful inhibition attempts. Our results confirm that (a) the latency of the error-related negativity (ERN; or Ne) component is too late to be associated with these online inhibitory mechanisms, and (b) instead, a frontal slow negative component with an earlier time course was associated with the implementation of online inhibition. These findings are consistent with single-cell recordings in monkeys showing that the supplementary motor area provides cognitive control signals to the primary motor cortex to exert online inhibition and in turn rectify the course of erroneous actions

Towards a new ITU-T recommendation for subjective methods evaluating gaming QoE

This paper reports on activities in Study Group 12 of the International Telecommunication Union (ITU-T SG12) to define a new Recommendation on subjective evaluation methods for gaming Quality of Experience (QoE). It first resumes the structure and content of the current draft which has been proposed to ITU-T SG12 in September 2014 and then critically discusses potential gaming content and evaluation methods for inclusion into the upcoming Recommendation. The aim is to start a discussion amongst experts on potential evaluation methods and their limitations, before finalizing a Recommendation. Such a recommendation might in the end be applied by non -expert users, hence wrong decisions in the evaluation design could negatively affect gaming QoE throughout the evaluation